Migrate RabbitMQ queues in ci-aio #1620

MoteHue · 2025-04-23T13:24:06Z

No description provided.

MoteHue · 2025-04-23T13:24:15Z

This relies on the following PR: #1619

.github/workflows/stackhpc-all-in-one.yml

Alex-Welsh · 2025-04-29T07:47:26Z

@MoteHue you might want to cherry-pick 3d847fa to fix the lint jobs

MoteHue · 2025-04-29T12:37:41Z

This needs to merge first: #1629

Alex-Welsh

LGTM if CI passes

MoteHue · 2025-04-30T14:30:39Z

Error in kayobe image building:

 #11 31.51 Error: 
#11 31.51  Problem: package gcc-11.5.0-5.el9_5.x86_64 from appstream requires glibc-devel >= 2.2.90-12, but none of the providers can be installed
#11 31.51   - conflicting requests
#11 31.51   - nothing provides glibc = 2.34-125.el9_5.3 needed by glibc-devel-2.34-125.el9_5.3.x86_64 from appstream
#11 31.51   - nothing provides glibc = 2.34-125.el9_5.3 needed by glibc-devel-2.34-125.el9_5.3.i686 from appstream
#11 31.51 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Alex-Welsh · 2025-05-01T07:33:04Z

Upgrade job fails with

  stderr: 'mv: cannot move ''/var/lib/docker/volumes/prometheus_v2/_data'' to ''/var/lib/docker/volumes/prometheus/_data'': Directory not empty'

The OVS jobs both fail with Neutron errors
The Nova API log says:

neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.

And Neutron Server says:

2025-04-30 15:58:06.057 27 INFO neutron.api.wsgi [None req-59fde41f-414b-4ea6-96f5-1c65e8f322b8 edd1dd0003a748f1b6f7f45aa138f47b 7f50da72d1a0478aaba23725a2a3879e - - default default] 192.168.33.2,192.168.33.3 "GET /v2.0/network-ip-availabilities HTTP/1.1" status: 200  len: 1298 time: 0.0107384
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation [None req-72faf510-e07d-4ff9-8f46-1e4e426652c4 7149963603b447caa9e1173800f7f834 1d056ff1fc0046efa37c0aca7e86e114 - - default default] GET failed.: AttributeError: 'NoneType' object has no attribute 'tags'
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/pecan/core.py", line 682, in __call__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/pecan/core.py", line 573, in invoke_controller
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 187, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     context_reference.session.rollback()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 135, in index
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return self.get(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 144, in get
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return {self.collection: self.plugin_lister(*lister_args,
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 223, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f_with_retry(*args, **kwargs,
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 187, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     context_reference.session.rollback()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1183, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return fn(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/db/db_base_plugin_v2.py", line 1682, in get_ports
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     items = [self._make_port_dict(c, fields, bulk=True) for c in query]
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/db/db_base_plugin_v2.py", line 1682, in <listcomp>
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     items = [self._make_port_dict(c, fields, bulk=True) for c in query]
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/db/db_base_plugin_common.py", line 257, in _make_port_dict
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     resource_extend.apply_funcs(
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/timeutils.py", line 323, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     result = func(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/resource_extend.py", line 91, in apply_funcs
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     resolved_func(response, db_object)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/services/tag/tag_plugin.py", line 54, in _extend_tags_dict
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     tag_db.tag for tag_db in db_data.standard_attr.tags
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'NoneType' object has no attribute 'tags'
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation

L3 agent logs say:

dhcp agent logs have:

oslo_messaging.rpc.client.RemoteError: Remote error: SubnetInUse Unable to complete operation on subnet 14855e14-668e-419d-aa9c-73b7d097265b: This subnet is being modified by another concurrent operation.

and again:

ERROR neutron.agent.dhcp.agent AttributeError: 'NoneType' object has no attribute 'tags'

MoteHue · 2025-05-01T12:11:55Z

Prometheus upgrade failure seems to be a bug in KA, I've raised it here: https://bugs.launchpad.net/kolla-ansible/+bug/2109784

Migrate RabbitMQ queues in ci-aio

ec95a9c

MoteHue requested a review from a team as a code owner April 23, 2025 13:24

product-auto-label bot added size: s workflows Workflow files have been modified labels Apr 23, 2025

MoteHue closed this Apr 23, 2025

MoteHue reopened this Apr 23, 2025

Alex-Welsh requested changes Apr 24, 2025

View reviewed changes

.github/workflows/stackhpc-all-in-one.yml Outdated Show resolved Hide resolved

.github/workflows/stackhpc-all-in-one.yml Outdated Show resolved Hide resolved

Alex-Welsh mentioned this pull request Apr 29, 2025

Sync stackhpc/2024.1 into master #1618

Open

MoteHue force-pushed the epoxy-rabbitmq-queue-migration branch from c83b800 to 7674d51 Compare April 29, 2025 09:56

product-auto-label bot added size: m and removed size: s labels Apr 29, 2025

MoteHue force-pushed the epoxy-rabbitmq-queue-migration branch from 7674d51 to bb7682d Compare April 29, 2025 10:00

Bump ansible-core version in lint jobs

c0d6b00

MoteHue force-pushed the epoxy-rabbitmq-queue-migration branch from bb7682d to c0d6b00 Compare April 29, 2025 12:37

product-auto-label bot added size: s and removed size: m labels Apr 29, 2025

Run upgrade prerequisites instead

9a4b5b3

Alex-Welsh previously approved these changes Apr 29, 2025

View reviewed changes

MoteHue closed this Apr 29, 2025

MoteHue reopened this Apr 29, 2025

Alex-Welsh mentioned this pull request Apr 30, 2025

Update to Prometheus v3 #1630

Merged

Merge branch 'stackhpc/master' into epoxy-rabbitmq-queue-migration

3fdb4f2

Alex-Welsh dismissed their stale review via 066b277 April 30, 2025 14:59

Alex-Welsh force-pushed the epoxy-rabbitmq-queue-migration branch from 066b277 to 3fdb4f2 Compare April 30, 2025 15:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate RabbitMQ queues in ci-aio #1620

Migrate RabbitMQ queues in ci-aio #1620

MoteHue commented Apr 23, 2025

MoteHue commented Apr 23, 2025

Alex-Welsh commented Apr 29, 2025

MoteHue commented Apr 29, 2025

Alex-Welsh left a comment

MoteHue commented Apr 30, 2025

Alex-Welsh commented May 1, 2025 •

edited

Loading

MoteHue commented May 1, 2025

Migrate RabbitMQ queues in ci-aio #1620

Are you sure you want to change the base?

Migrate RabbitMQ queues in ci-aio #1620

Conversation

MoteHue commented Apr 23, 2025

MoteHue commented Apr 23, 2025

Alex-Welsh commented Apr 29, 2025

MoteHue commented Apr 29, 2025

Alex-Welsh left a comment

Choose a reason for hiding this comment

MoteHue commented Apr 30, 2025

Alex-Welsh commented May 1, 2025 • edited Loading

MoteHue commented May 1, 2025

Alex-Welsh commented May 1, 2025 •

edited

Loading